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Abstract 

This paper presents the rationale and methods for a randomized controlled evaluation of web-based training in 
motivational interviewing, goal setting, and behavioral task assignment. Web-based training may be a practical and 
cost-effective way to address the need for large-scale mental health training in evidence-based practice; however, 
there is a dearth of well-controlled outcome studies of these approaches. For the current trial, 168 mental health 
providers treating post-traumatic stress disorder (PTSD) were assigned to web-based training plus supervision, web- 
based training, or training-as-usual (control). A novel standardized patient (SP) assessment was developed and 
implemented for objective measurement of changes in clinical skills, while on-line self-report measures were used 
for assessing changes in knowledge, perceived self-efficacy, and practice related to cognitive behavioral therapy 
(CBT) techniques. Eligible participants were all actively involved in mental health treatment of veterans with PTSD. 
Study methodology illustrates ways of developing training content, recruiting participants, and assessing 
knowledge, perceived self-efficacy, and competency-based outcomes, and demonstrates the feasibility of 
conducting prospective studies of training efficacy or effectiveness in large healthcare systems. 

Keywords: Post-traumatic stress disorder (PTSD), Cognitive behavior therapy, Randomized controlled trial, 
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Background 

As a consequence of the wars in Iraq and Afghanistan, 
increasing numbers of active duty personnel are being 
exposed to deployment-related traumatic experiences 
resulting, in some instances, in the development of post- 
traumatic stress disorder (PTSD) and other mental 
health problems. Moreover, as a result of Department of 
Defense (DoD)- and Veterans Health Administration 
(VH A) -mandated universal screening for PTSD and mili- 
tary sexual trauma (MST), increasing numbers of service 
men and women and veterans in need of mental health 
support are being identified. Prevalence estimates for 
PTSD range from 12% to 20% of returnees from Iraq, 
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and 6% to 11% of returnees from Afghanistan [1]. Dis- 
semination and effective delivery of short-term, effica- 
cious, evidence-based treatments for PTSD is an urgent 
priority to address this growing public health problem 
and to prevent the development of costly chronic mental 
health disorders among returning veterans from the con- 
flicts in Iraq and Afghanistan [2]. Such treatments 
should also be made available to those experiencing 
PTSD symptoms as a result of involvement in previous 
wars. 

The internet potentially provides an ideal vehicle for 
cost-effective training of mental health providers and a 
potentially effective means to address challenges asso- 
ciated with large-scale face-to-face training initiatives [3]. 
Once established, online training programs can be deliv- 
ered on a continuing basis to large numbers of providers 
with little or no incremental cost. Moreover, internet-based 
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methods can overcome logistical barriers by virtue of their 
ease of accessibility in the office or home. Despite these 
established advantages, questions remain about the imme- 
diate and long-term effectiveness of internet-based train- 
ing, particularly in relation to mental health interventions. 

Research provides strong support for the effectiveness 
of mental health interventions based on cognitive- 
behavioral therapy (CBT) for most psychological pro- 
blems that bring veterans into care (PTSD, depression, 
anxiety, and substance abuse), leading to the highest pri- 
ority for implementation of CBT in the VHA and other 
healthcare systems [4,5]. In response to the urgent need 
for large-scale training, several system-wide initiatives 
have been launched inside the VHA and in other public 
programs to disseminate CBT interventions for psycho- 
logical problems [5,6]. Despite widespread acceptance of 
the clinical relevance and need for these types of inter- 
ventions, there is no consensus on best practices for 
accomplishing large-scale dissemination, necessary ele- 
ments of effective training, or methods of assessing suc- 
cessful outcomes of training [5]. Current research does 
not sufficiently inform us regarding the effectiveness of 
web-based training methods either as stand-alone meth- 
ods, or with supplementation via individual supervision 
with qualified experts. Post-training supervision (whether 
face-to-face or remote) is hypothesized to significantly 
influence behavior of clinician trainees [7], although the 
empirical support for this assumption is not robust. 

The gap between evidence-based interventions and 
current practice in service delivery has been noted in pre- 
vention, chronic care, and many aspects of clinical practice 
in healthcare in diverse settings and across many disci- 
plines [8-10]. One shortcoming is that many treatment 
programs fail to adequately identify core assessment and 
treatment techniques and provide training in those techni- 
ques [11,12]. Face-to-face training, while effective in some 
areas of intervention, is too resource-intensive to provide 
on the scale required to reach community-based programs 
[13]. Other lines of research are attempting to improve the 
transportability of existing CBT interventions by tailoring 
or optimizing them for delivery in specific systems or for 
specific audiences, such as with children [14] or veterans 
[15], by increasing flexibility in implementation [16], and 
by better integrating cross-cultural awareness into educa- 
tion programs [17]. It is becoming commonplace to in- 
corporate web-based training as one component of 
dissemination programs for providers in many professions 
[7,18-21]. The aim of the newest generation of studies is to 
identify the combinations of training, supervision [22], and 
support [23] that are necessary and sufficient to maintain 
good fidelity to evidence-based CBT techniques. 

In this paper, we describe the development and testing 
of an internet-based training program to deliver compo- 
nents of CBT for PTSD, as a stand-alone training and 



with a group telephone supervision process designed to 
supplement the training and improve participants' deliv- 
ery of CBT techniques. 

Methods 

Study rationale 

The overall objective of our study is to conduct a pro- 
spective, randomized controlled trial to evaluate the 
feasibility, implementation, and effectiveness of internet- 
based training in CBT techniques, including a standar- 
dized telephone-delivered supervision method. Second- 
ary aims of the study are: to develop a module-based 
online training program in CBT techniques that is suffi- 
ciently engaging to encourage maximal participation; to 
recruit a large, diverse sample of mental health providers 
for participation in training research under 'real world' 
conditions; to design and administer web-based surveys 
to assess knowledge acquisition and changes in perceived 
self-efficacy; to design, implement and evaluate a struc- 
tured method for telephone-based case supervision; and 
to develop and implement a standardized patient behav- 
ioral observation assessment methodology for evaluating 
skills performance during simulated treatment delivery. 

Rationale for selection of core training modules 

The CBT techniques were divided into three training 
modules: motivational interviewing; goal-setting; and be- 
havioral task assignment. These modules were selected 
for multiple reasons. First, all three modules focus on 
the essential need to prompt behavior change in those 
seeking treatment services. Second, they represent be- 
havior change skills that can be trained with relatively 
brief investment of time and effort for potentially large 
yield in clinical performance and outcomes. These core 
techniques and practices are typically embedded in 
multi-session CBT package interventions, which made 
conversion to web-based training format readily feasible 
for the current study. Third, the modules we selected 
focus on generic, multi-purpose CBT techniques, which 
can be broadly applied across a wide range of behavior 
change targets. Finally, these techniques can be inte- 
grated by diverse providers into ongoing therapy sessions 
flexibly and immediately. The practices to be dissemi- 
nated were also considered to 'fit' well within existing 
structures of PTSD care within VHA (e.g., time-limited 
approaches), an important consideration in dissemin- 
ation, and to reflect expressed training interests among 
PTSD treatment providers who participated in focus 
groups. Generally, the three techniques have not been 
studied as stand-alone interventions for PTSD, and they 
are not designed to represent comprehensive PTSD 
interventions. Motivational interviewing has been used 
successfully to increase motivation to change a range of 
behaviors that co-occur with PTSD, whereas goal-setting 
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and behavioral task assignment are components of exist- 
ing evidence-based PTSD protocols (e.g., prolonged ex- 
posure and cognitive processing therapy), as well as 
interventions for other trauma-associated problems. 

Experience within VHA PTSD treatment programs indi- 
cates that many veterans fail to engage in treatment, drop 
out after attending one or two sessions, or are reluctant to 
fully participate (e.g., attend group sessions). Cognitive- 
behavioral motivational interviewing using the widely 
tested methods of Miller and Rollnick [24] was therefore 
selected as one empirically-supported technique that pro- 
viders could employ to help motivate veterans to partici- 
pate in the treatment process. This technique also helps 
providers respond to patient ambivalence about changing 
key PTSD-related behaviors, such as substance abuse, mal- 
adaptive weapon storage, and violent behavior, all preva- 
lent among returning service members with PTSD [25] . 

Goal-setting was selected as a core technique because 
delivery of CBT interventions is predicated on a process 
of mutual patient-provider negotiation of treatment 
goals. Nevertheless, in routine clinical care, this mutual 
and systematic negotiation of short-term treatment goals 
often does not occur. Instead, patients are offered a set 
of standard treatment options (e.g., participating in a 
PTSD education class, receiving a stress management 
intervention). It is likely that creation of individualized 
goals will increase patient engagement by maximizing 
the personal relevance of treatment. Such a client- 
centered approach also increases the likelihood that the 
patient will attempt behavior change and attend treat- 
ment sessions. For the mental health provider, such goal- 
setting helps to ensure 'customer-focused' treatment. 

A core technique in CBT interventions is behavioral task 
assignment that facilitates transfer of training out of the 
office into the real world environment of the patient. 
Patients who work to change their behavior outside of the 
counseling session can be expected to accomplish more 
significant change. They can achieve concrete benefits and 
increase their perceived self-efficacy in coping with their 
life challenges. As patients complete self-monitoring 
assignments via handouts and forms, they can more sys- 
tematically learn about themselves and provide the clinician 
with clearer accounts of between- session behaviors. The 
provider reinforces successful task performances and works 
with patients in a variety of ways to reduce non-adherence. 
Thus, this CBT technique was selected for web instruction 
because it is versatile and can increase the impact of rou- 
tine clinical interactions, and because many providers are 
not currentiy using behavioral task assignments. 

Format of the training modules 

The CBT training modules were designed to be brief, 
with each requiring approximately 80 min online to 
complete. Components of the training modules include: 



1. Learning objectives that outline the expectations for 
each module; 

2. Interactive components that illustrate key points and 
demonstrate real-world application of the techniques 
(e.g., via audio-streamed modeling); 

3. Downloadable handouts that summarize content for 
providers; 

4. Downloadable patient education materials that 
accompany the provider content; and 

5. A final knowledge quiz enabling participants to 
qualify to receive continuing education (CE) credits. 

Case examples were also carefully developed to illus- 
trate practical aspects of implementation and challenges 
commonly faced by providers. In essence, these cases 
provide authentic examples of how patients actually 
present. 

Content development 

Development of the training modules required several 
steps to ensure that the content was created in conjunc- 
tion with the latest behavioral evidence, that the infor- 
mation delivered was balanced, and that the participants 
were provided with necessary information to be consid- 
ered trained in these core aspects of CBT. Experts in the 
techniques (motivational interviewing, goal-setting, and 
behavioral task assignment) were consulted to provide 
content oversight on the core modules. Once the module 
drafts were complete they were reviewed by study staff 
and by additional experts in CBT who had not been 
involved in the development of the modules. This im- 
portant step ensured an independent review of the con- 
tent. Once modules were reviewed and approved, a final 
script was created and reviewed a final time to ensure 
that the integrity of the initial content document was 
maintained after adding interactive elements and format- 
ting text for the web. 

Each module included interactive audio elements in 
which a case interaction was presented to demonstrate 
appropriate and inappropriate approaches to technique 
delivery (See Figure 1). Course modules also included 
interactive exercises (Figure 2) that reinforced key con- 
cepts listed in the learning objectives and presented in- 
formation in an easily accessible PDF format for 
worksheets and other training-related documents. 

While content development was being conducted by 
the authors, the website graphic design was created, 
reviewed, and approved, creating a distinctive 'look and 
feel.' Formatting of interactive elements (audio, PDF 
styles, and animation) was conducted and all materials 
were developed according to guidelines to make elec- 
tronic information accessible to people with disabilities 
(508 compliance) available at the time of the study. Sev- 
eral quality assurance steps were conducted to ensure 
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Figure 1 Example of audio dialog with photo slideshow. 



that the program functioned properly. Testing was con- 
ducted on each completed module to ensure that all con- 
tent flowed properly, and the entire site was subject to a 
formal quality review prior to launch. 

Telephone supervision process 

Supervision was aimed primarily at assisting providers in 
learning and implementing new techniques acquired 
from the web-based CBT training. During the supervi- 
sion sessions, providers were guided in implementing 



specific techniques in a routine manner in their VHA 
treatment setting and in fully grasping the theoretical 
and practical aspects of these new techniques. Studies of 
the supervision process have suggested that cognitive- 
behavioral supervision in which supervisors monitor the 
clinician trainee, model competent technique delivery, 
give specific instructions, set goals, and provide feedback 
on performance are associated with benefits to clinician 
trainees [26]. Supervisors were encouraged to apply the 
same core CBT techniques of motivational interviewing, 
goal-setting, and behavioral task assignment in their own 
work with clinician trainees during supervision phone 
calls. That is, supervisors were asked to model the tech- 
niques during their interactions with clinician trainees. 

To translate to larger-scale training efforts, a system of 
supervision must be feasible to deliver, brief enough to 
encourage clinician trainee participation and completion, 
and sufficiently comprehensive to ensure learning of new 
techniques. Given the prohibitive costs of developing 
local supervision capability for all VHA locations to en- 
sure access to face-to-face supervision, if supervision is 
to be delivered at scale, it will likely need to be offered 
at a distance via telephone or videoconferencing. The 
model of telephone supervision described here was 
intended to represent a 'middle-ground' approach that 
might make a meaningful impact on participant skills 
and implementation, while limiting costs and staffing re- 
source demands of supervision. An intensity of five to 
six weekly supervision sessions, each lasting a maximum 
of 60 min, was selected on the assumption that a less in- 
tensive model would have limited impact, while requir- 
ing more sessions might place excessive demands on 
participant time. Because VHA providers most often 
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I can t figure out why I feel so different 
than I did before the war. 

I'm not going to the VA: I'm not like 
those Vietnam vets. 

I'm trying to practice breathing and 
relaxing when I'm getting tense but it's 
really hard to do. 

One of the ways I stay clean and sober 
is to keep up with my AA meetings. 



Figure 2 Example interactive exercise. 
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schedule for 50-minute treatment sessions, these time 
frames fit well with scheduling for participants. 

Supervisors were selected from National Center for 
PTSD Dissemination and Training Division staff. They 
had doctorates in clinical psychology and were familiar 
with cognitive-behavioral psychological theory and treat- 
ment methods. Several had extensive experience in con- 
ducting training workshops and were experts in specific 
CBT interventions {e.g., prolonged exposure, acceptance 
and commitment therapy). 

To ensure a structured supervision process, a manual 
for providing clinical supervision via telephone was 
developed. Training for supervisors included: completion 
of the web modules; review of important theoretical and 
empirical background information about CBT; comple- 
tion of a one-day training including a refresher on the 
three core CBT techniques; training in supervision meth- 
ods via lectures and video illustrations; and practice in 
giving feedback on therapy sessions. Supervisors also 
participated in bi-monthly teleconference calls to obtain 
regular feedback on their supervision of newly trained 
CBT therapists. 

Six primary supervisors and two back-up supervisors 
were available to conduct calls. Supervision was sched- 
uled using a cohort design, in which each supervisor was 
expected to complete six small-group calls over six con- 
secutive weeks with the same cohort of clinician trainees. 
Each cohort comprised one to four clinician trainees and 
new groups were started every week. Order of supervisor 
participation was chosen in advance, and each newly- 
formed cohort was assigned to the next-in-line super- 
visor. Back-up 'on-call' supervisors were asked to cover 
for primary supervisors unable to make their regularly 
scheduled calls due to illness or other unforeseen 
problems. 

Immediately after completing the web training, partici- 
pants assigned to the web-based training plus supervi- 
sion group received an email description of the format 
and procedures of supervision, self-monitoring home- 
work forms, and confirmation of their first consultation 
session. 

Clinician trainees were asked to self-monitor their ap- 
plication of the core techniques via systematic use of a 
CBT session checklist created for the project. This self- 
monitoring activity was expected to help focus clinician 
trainees and supervisor attention on core aspects of the 
techniques. A CBT session checklist was the primary 
means for assessing adherence to elements of the techni- 
ques during the study, and both clinician trainees and 
supervisors were expected to have a copy of the form in 
front of them at all times during supervision calls. Clin- 
ician trainees were expected to submit these checklists 
to their supervisors prior to each supervision call via fax 
or email. Supervisors were expected to review clinician 



trainees' forms before each supervision call and use the 
information to set a focused agenda for the session. As 
an additional teaching aid, clinician trainees were asked 
to complete some of the same homework forms that are 
typically assigned to patients as a way to track their own 
clinical behaviors/skills. A 'Monitoring Your Goals' form 
was used for general tracking of clinical goals over the 
week between supervision calls and a task assignment 
log was used to track specific behaviors (e.g., the clinician 
trainee's use of the motivational interviewing 'personal 
ruler' technique). To meet the criterion for completion 
of the supervision process, participants had to complete 
three or more of the possible six supervision sessions. 

Recruitment and randomization 

PTSD treatment providers working in active treatment 
settings were recruited to test the continuing education 
course modules. To be eligible for study participation, a 
provider had to be employed full-time as a VHA provider 
in a mental health clinic, and be treating patients with 
PTSD at the time of recruitment. VHA protocol prohi- 
bits compensating staff members for research participa- 
tion; however, in addition to the opportunity to learn the 
new techniques presented in the training, all participants 
were offered the opportunity to earn four professional 
continuing education (CE) credits. Those assigned to the 
training-as-usual (control) group were able to complete 
the training and receive continuing education credits 
after their participation in the study was completed. 

Because of restrictions on participant compensation, 
there was an initial concern that, for busy providers, re- 
cruitment might be challenging. Thus, careful attention 
was paid to recruitment activities and the opportunity to 
participate in the research project was announced in sev- 
eral forums. Study fliers (Figure 3) were produced and 
distributed at two VHA mental health conferences and 
at several training events (e.g., National Center for PTSD 
Clinical Training Program). Four email lists of mental 
health providers across the country were compiled and 
sorted randomly to avoid any potential biases. To 
minimize selection bias, during each week of the recruit- 
ment phase, approximately 50 providers per week were 
contacted from the four lists equally, and invited to par- 
ticipate. Providers responding to the announcements 
accessed the study website, where they were introduced 
to the study process, completed brief online questions 
that screened for eligibility, and if eligible, completed an 
online consent process. 

Once consented, participants were randomly assigned 
to one of three study conditions: web-based training plus 
supervision, web-based training, and training-as-usual 
(control). Participants who completed the baseline as- 
sessment, including the simulated patient interview, were 
stratified by self-reported previous CBT experience and 
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Online Training in CBT Skills for PTSD Treatment: 

Research Project Currently Enrolling 



Visit our website for more information or to sign up! 



Useful Training! 

•CEUs 'Self-paced - Interactive 
•Focused on 3 key components of CBT 



Figure 3 Recruitment postcard. 



randomized to one of the three training groups. We used 
permuted blocks of nine participants within each 
stratum. The control group clinician participants were 
free to participate in any training activities they would 
otherwise receive, including local continuing education 
activities, conference participation, or other formal train- 
ing programs taking place in the VA. Participants were 
stratified on the basis of self-rated degree of expertise in 
cognitive-behavioral therapy (low = none or beginner; 
high = intermediate, advanced, or expert) and were ran- 
domized within strata to one of the three training groups 
(See Table 1). 

In this study, a web-based screening instrument was 
developed and made available through a recruitment 
registration website announced on flyers and in recruit- 
ment emails. Participants accessed the online screener 
that determined eligibility for the study. Individuals were 
excluded if they: were not a full-time VHA or Vet Center 
employee; were not a clinical psychologist, social worker, 
psychiatrist, readjustment counselor, mental health 
counselor, substance abuse counselor, nurse, psychology 



Table 1 Stratified random group assignment 





Self-rated CBT expertise 


Random Group Assignment 


Low 


High 


Training-as-usual 


19 


37 


Web-based training only 


21 


36 


Web-based training +expert supervision 


19 


36 



trainee, or a trainee classified as 'other'; were not cur- 
rently counseling/treating patients with PTSD as part of 
their VHA or vet center clinical work; or had previously 
completed VHA-offered training in cognitive-behavioral 
therapy for depression (whose content domains over- 
lapped significantly with web module content). 

Those eligible were directed to an online informed 
consent process. If consent was completed (requiring 
two online signatures), the baseline assessment instru- 
ment was made available. Those not eligible were 
thanked and informed that they were not eligible for the 
study. A web-based baseline assessment was adminis- 
tered to determine baseline knowledge and perceived 
self-efficacy. In addition, participants had to schedule 
and complete a standardized patient interview (SPI, Time 
1, discussed below). Both the baseline assessment and 
SPI were required prior to randomization and access to 
the web training. Randomization to one of the three 
groups occurred once the participant was identified as 
having completed the SPI. This process is illustrated in 
Figure 4. 

If the participant was randomized to either of the web- 
based training groups, a user name and password was 
automatically generated and sent to the participant's 
email. Participants in the two web-based training groups 
had a five-week window to complete the web training 
modules at their own pace and in any order before being 
prompted to complete the midpoint assessment. When 
the midpoint assessment was due, access to the web 



Ruzek et al. Implementation Science 2012, 7:43 
http://www.implementationscience.eom/content/7/1/43 



Page 7 of 14 




Screening 
& 

Registration 



IF ELIGIBLE 
► 



Baseline 
Assessment 
Questionnaire 



Figure 4 Screening and enrollment process. 
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training was suspended until the assessment was com- 
pleted. After completion, the web training was again 
made available. The midpoint assessment included an 
evaluation of the training and assessment of knowledge 
(which also served as the CE quiz). This midpoint assess- 
ment also served as a trigger for the web-based training 
plus supervision group to begin supervision calls. Partici- 
pants randomized to web-based training (without supervi- 
sion) were allowed continued access to the training 
materials but had no other additional contact until the end 
of the six-week supervision period. Participants rando- 
mized to web-based training plus supervision began sched- 
uling regular supervision calls over a six-week period. At 
the completion of the six-week supervision period, partici- 
pants in all three groups were contacted by email to begin 
scheduling the second SPI (Time 2). Immediately following 
completion of the second SPI all participants were sent an 
email directing them to the web-based post-training as- 
sessment. Completion of the post-training assessment con- 
cluded study participation. At this point, training-as-usual 
(control) group participants were provided access to the 
training and were offered the opportunity to complete the 
CE questionnaire for credits. This overall process is illu- 
strated in Figure 5. Ethical review and approval for the 
study was provided by the Stanford University and NERI 
Institutional Review Boards and the VA Palo Alto Health 
Care System Research and Development committee. 

Assessment methods 

Evaluation strategy and procedures 

Assessment of effectiveness of online training is intrin- 
sically challenging. Assessment methods in large health- 
care systems must be easily implemented, relatively brief 
in duration, and scalable at modest cost to reach large 
numbers of geographically-distributed providers. In the 



current study, online self-report questionnaire methodology 
provided a relatively simple way of assessing results of train- 
ing for a large number of mental health providers. Three 
core domains of outcome were measured: knowledge 
assessed through multiple-choice tests; self-reports of per- 
ceived self-efficacy in technique application; and to assess 
implementation of CBT techniques, self-reported frequency 
of application with PTSD patients. The content of web- 
based data collection and the assessment time points are 
shown in Figure 6. 

While self-report instruments can assess knowledge, 
perceived self-efficacy, and participant-perceived degree of 
use of CBT techniques, they are limited in their ability to 
examine actual skills development. The latter outcome is, 
of course, especially important in evaluating the success of 
training. To assess actual mastery of therapy skills, our pri- 
mary outcome measure was based on simulated treat- 
ment-delivery interviews with standardized patients. 
Standardized patients have been used widely in medical 
settings to teach and measure skills of assessment and 
diagnosis [27,28]. This methodology has rarely been used 
to examine intervention skills. However, a small but grow- 
ing literature exists for the use of standardized patients in 
mental health training, particularly to evaluate clinical 
interviewing for substance abuse [29,30]. We conducted 
the standardized patient interview sessions over the phone, 
to facilitate assessment of mental health providers located 
across the country. This combination of online and tele- 
phone assessment methodologies was deemed practical to 
deliver and complementary in terms of information yield. 

Standardized patient (SP) interview methodology 

Measurement of participant performance in simulated 
sessions with the standardized patient was used to assess 
actual skills acquisition. The goal was to create an 
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Figure 5 Study design. 
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Figure 6 Web-based research measures. 



assessment protocol that ensured standardization of 
stimulus conditions while permitting actors in the stan- 
dardized patient role to respond with considerable flexi- 
bility depending upon which of a wide range of provider 
behaviors occurred. The standardized patient interviews 
were conducted at baseline and follow-up for all 
participants. 

Participants received email orientation materials 72 h 
prior to their interview. This orientation included a brief 
description of procedures for the standardized patient 
interview, description of the clinical background of the 
patient, information about the time and date of the 
phone interview, and the phone number at which they 
would be contacted. 24 h prior to the scheduled inter- 
view, the actor was instructed to call the participant and 
remind him or her to read the materials and confirm 
when the interview would take place. If the participant 
did not answer this call, the actor was instructed to leave 
a voicemail or a message with the receptionist if voice- 
mail was not available. 

Simulated interviews were conducted during 50-minute 
telephone calls. The 50-minute sessions were divided into 
a five-minute introduction/orientation followed by three 
15-minute segments (one for each intervention). We 
defined a segment as a portion of the standardized patient 
interview that could be rated according to study criteria. 

To ensure a uniform set of stimulus conditions, a scripted 
approach to the simulated interviews was developed. Upon 



initiation of the interview, scripted actor statements were 
used to orient the participant to the process, and ensure 
readiness to participate in the interview {e.g., to ensure that 
the participant had read the patient background material, 
and was located in a quiet environment with relative protec- 
tion from interruption). Standardized scripts were also used 
to open the three segments. In this way, it was possible to 
convey standard presentations across conditions and actors 
and provide participants with standardized opportunities 
to demonstrate appropriate techniques. Because of the po- 
tential for variation in participant behavior, conditional 
responses (Figure 7) were developed to guide actor response 
after the opening statements. The rules were kept relatively 
simple in order to avoid overloading the actor with per- 
formance demands. 

Effective actor selection and training were critical to 
ensure authenticity of the interaction. To ensure that 
actors were familiar with the clinical presentation of 
PTSD in veteran patients, clinical psychology graduate 
students were interviewed for potential participation. 
Those selected had typically worked in a VHA practicum 
setting and performed well on a practice role play that 
was part of the selection process. 

Actors were given a manual describing the interview 
procedures. The manual included background informa- 
tion about the patient they would portray, an overview 
of the procedures for the interview, a telephone script 
for the one-day reminder phone call about the interview, 
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If CT [participant] does all or part of balance sheet exercise, respond as appropriate, your 
answers are the following: 



Advantages of things 
staying the same? 

"Don't have to drive so 
much." 

"Don't have to get close to 
my wife." 



Disadvantages of things 
staying the same? 

"My wife wouldn't be happy 
with me." 

"Might not be able to deal 
with it on my own." 



Disadvantages of 
changing? 

"I would have to drive 45 
minutes each way." 

"I will have to get used to 
talking about what I'm 
feeling." 



Advantages of 
changing? 

"Maybe counselors can 
help me. 

My wife will be happy that 
I'm getting help." 

"I'll feel like I'm doing 
something. 

It'd be great to be back in 
touch with other Vets who 
know where I'm coming 
from." 



If CT [participant] asks follow-up questions to the balance sheet exercise, respond as 
appropriate, your answers are the following: 



Did you find that you're discouraged 

about trying again because of your " Yes - a '' ttle Decause 1 d ' d treatment before and I'm back here again." 

past history? 



What would it really take for you to ., know , have t0 set my mind t0 „ and rea||y work on myse|f „ 

change? 



Figure 7 Example conditional responses from standardized script. 



the interview script (including general instructions and 
scripted statements), conditional responses, and tech- 
nical instructions for audio recording the interview. 
Background information about the patient was intended 
to provide the actors with enough information to 
complete their interactions with participants in a manner 
consistent with natural therapeutic conversation and 
allowing flexible responses to questions. Background de- 
scription included details of personal history, medical 
history, symptom display, reasons for seeking help, per- 
sonality (i.e., response style), affective presentation, and 
general disposition; these areas were covered in anticipa- 
tion of possible questions and statements produced by 
participants. Actor training sessions were held in close 
time proximity to the conduct of the interviews. Each 
actor completed a two-day intensive in-person training 
workshop where they were given an overview of the 
study protocol, watched video footage of veterans with 
PTSD, participated in a question-and-answer session 
with a veteran who had recently graduated from a local 
inpatient PTSD program, reviewed the protocol for each 
segment of the interview, and then participated in nu- 
merous role-plays. Actor trainees were given significant 
opportunity to practice each of the techniques in the 
simulated interview and receive feedback on their perfor- 
mances. They worked as a team during practice sessions 
to observe one another's performances and decide if any 
alterations needed to be made to the standardized pa- 
tient protocol. 

After it was judged that the standardized patient proto- 
col and actor performances were adequate, 'dress 



rehearsals' were conducted in which the actors called 
study personnel who played the role of study partici- 
pants. Actors were required to demonstrate their fidelity 
to the interview structure as well as ability to successfully 
record the session and deliver the recording to the study 
manager. 

The script developed for the standardized patient 
interview included instructions for the conduct of the 
interview, and an outline of the clinical details and 
instructions for each of the three interview techniques 
(motivational interviewing, goal-setting, behavioral task 
assignment). Furthermore, opening statements were 
scripted to begin each of the segments. In the case of 
motivational interviewing, two additional scripted state- 
ments were included to provide opportunities for partici- 
pants to respond to deliberate change and counter- 
change statements. Conditional responses were designed 
to facilitate anticipated techniques taught in each of the 
respective modules. To maintain consistency, the actors 
were trained to identify and respond to some of the 
specific techniques with scripted responses. For example, 
in the motivational interviewing module, participants 
were taught to use a decisional balance sheet exercise 
(Figure 8), and actors were trained on the conditional 
responses illustrated in Figure 7. Actors were blind to 
the condition to which the participant was assigned. 

Rating of simulated interview performances 

Raters were graduate student research staff not deliver- 
ing the intervention, all of whom were masked to study 
condition and time point of assessment. Five practice 
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Concept Check 



Using a Decisional Balance Worksheet 

A PTSO patient keeps a loaded weapon in his bedroom. In the decisional balance exercise, he 
came up with the following pros and cons. Drag the statements beta'// to the boxes where they 
fit best. 



Statement 1 of 23: 



My wife will be happier. 



1 Advantagos of things staying tho 
same 


3 Not so good things about changing 


i 




2 Not so good things about remaining 
the same 


4 Advantagos or benefits of changing 







Figure 8 Decisional balance interactive exercise. 



transcrips were rated independently and discussed as a 
group to identify points of convergence and divergence. 
Decisions about rating disagreements were reached and 
clarifications were added to the rating guide. Once rating 
began, 20% of all transcripts were randomly selected to 
be re-rated by a second rater. Raters were masked as to 
which transcripts were being re-rated. 

For each of the CBT techniques, raters scored tran- 
scripts based on the degree to which specific technique- 
related participant behaviors were observed. Each behav- 
ior was rated as '0' (Does Not Employ), '1' (Partially 
Employs), or '2' (Consistently Employs). Behavioral cri- 
teria for the three rating levels guided raters. For motiv- 
ational interviewing, goal-setting, and behavioral task 
assignment, 17, 8, and 15 behaviors, respectively, were 
rated. Examples of techniques for behavioral task assign- 
ment were 'frame as an experiment' and 'generate poten- 
tial tasks.' For each of the three techniques, raters also 



scored 'overall degree of collaboration' (0 to 4) and 'over- 
all effectiveness' with the techniques (0 to 4). 

Statistical analysis plans 

Primary analyses The primary outcome of the study 
intervention is the performance of participants in dem- 
onstrating CBT techniques during the SPI assessment at 
post-training. Summary scores for each of the CBT tech- 
niques will be constructed from the ratings of the SPIs 
and the properties of the scores evaluated. The analyses 
to be performed will compare the performance of parti- 
cipants in the training-as-usual (control) and web-based 
training groups using the analysis of covariance 
(ANCOVA) comparing the average post-test scores 
among the three training groups, adjusting for baseline 
scores [31]. We hypothesize that participants in the web- 
based training groups will exhibit greater improvement 
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in skills performance than those in the training-as-usual 
(control) group. We also hypothesize that web-based 
training plus supervision will be associated with greater 
skills improvement than web-based training alone. If the 
null hypothesis of no difference among the three groups 
is rejected, we will compute pre-planned comparisons of 
the scores between the individual groups adjusting for 
multiple comparisons. All participants will be analyzed 
in the training group to which they were randomly 
assigned, regardless of their compliance with the 
assigned training method and the assessments. Partici- 
pants who do not complete the post-training SPI will be 
considered to have the same value at post-test as at base- 
line. To assess sensitivity of our results to potential re- 
sponse bias due to more 'successful' trainees completing 
the follow-up assessment, we assumed that non-completers 
made no change from baseline (last observation carried 
forward). This is a commonly used method of imputation 
when doing this type of sensitivity analysis. 

Analyses focusing on the intent-to-train sample will 
also be repeated for the sample of participants who com- 
pleted 'therapeutic doses' of the training interventions. 
To meet the criterion for completion of the web portion 
of the training, a participant will have to complete 75% 
of each of the three modules. To complete the web- 
based training plus supervision process, a participant will 
have to complete 75% of each of the online modules and 
attend three or more of the supervision sessions. 

Secondary analyses We also hypothesize that providers 
in the web-based training groups will show significantly 
greater improvement in CBT-related knowledge, per- 
ceived self-efficacy, and self-reported implementation of 
CBT techniques than those in the training-as-usual (con- 
trol) group. The baseline and post-test assessments con- 
sist of multiple choice knowledge items, perceived self- 
efficacy, and practice behavior variables. As with the SPI 
scores, summary scores will be calculated from the indi- 
vidual questions. Treatment groups will be compared 
using ANCOVA on the post-test scores controlling for 
the baseline scores with follow-up comparisons as 
described above. Moderator analyses are not planned 
given the use of a randomized trial for testing the pri- 
mary and secondary hypotheses of the study. 

Power analysis Power calculations were computed 
based on the estimated percentage of participants who 
would successfully implement the techniques as assessed 
by the SPI. These calculations indicated that 40 partici- 
pants per group (total n = 120) would provide 79% power 
to detect a rate of successful implementation of 48% in 
the web-based training (without supervision) group com- 
pared to 15% in the training-as-usual (control) group, 
and 91% power to detect a rate of 54% in the web-based 



training plus supervision group compared to 15% in the 
training-as-usual (control) group, both at the 2.5% sig- 
nificance level. 



Results and discussion 

In this paper, we present evidence for the effectiveness of 
our study recruitment procedures. As shown in the flow- 
chart of recruitment and study completion in Figure 9, 
we exceeded our recruitment goals. The study protocol 
called for recruitment of an approximate target of 135 
mental health providers; in fact we were able to 
randomize 168 participants for the study, ahead of the 
planned recruitment pace. In keeping with the overall 
composition of VHA mental health providers, our sam- 
ple was predominantly white (79%), female (70%), and 
had post-graduate training in psychology, social work, 
nursing or psychiatry. The average age was 49, and more 
than 90% had post-graduate qualifications (Master's or 
Doctoral level). 

Recruitment was scheduled to take place over the 
course of 12 months; however, due to the rapid success 
of the recruitment process, full recruitment was achieved 
in nine months, with an average recruitment rate of 19 
participants per month. Recruitment began to exceed 
targeted numbers in month two of the project, and con- 
tinued to exceed targets during the successive months. 
This is a key indicator of both perceived need for web- 
based CBT training and feasibility of recruitment. Effects 
of the training interventions on the range of outcome 



Screened 
Ineligible 
55 



Web Training 

N=57 



Screened Eligible 

298 

I 

Completed Baseline 
Questionnaire 
SIM Patient Interview 

168* 



Randomization 



Web Training + 
Consultation 



N=55 

Completed Post Training Assessment 

46 42 



Training As Usual 

N=56 



51 



* 1 of these interviews lost to technical error 
** 2 of theses interviews lost to technical error 



Figure 9 Flowchart of participant recruitment and study 
completion. 
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variables will be the focus of presentation in our forth- 
coming papers. 

Discussion 

There are few randomized, controlled trials of training 
interventions for best practices in mental health, and still 
fewer outcome studies of web-based training methods. 
While web-based interventions have been shown to be 
effective in changing multiple health behaviors in diverse 
patient populations [32], few studies to date have exam- 
ined their use in the context of training for trauma inter- 
vention among practitioners. We present here the 
design, rationale, and methods for the first randomized, 
controlled trial of web-based training for mental health 
providers providing clinical care to veterans with PTSD. 
The central question to be addressed is whether web- 
based training, with or without individual supervision, 
may provide an effective means to train increasing num- 
bers of mental health providers in relevant, evidence- 
based clinical techniques. The lack of controlled out- 
come studies of evidence-based training in mental health 
practice is of special concern, given the urgent need for 
increased clinical services in this area [3]. Web-based 
training interventions, in particular, offer substantial po- 
tential for cost-effective application to a broad, multi- 
disciplinary audience of mental health providers. 

The training component of the study is innovative in 
being entirely web- and telephone-based, and therefore 
easily accessed and completed by participants. Impor- 
tantly, it delivers instruction in core techniques (e.g., mo- 
tivational interviewing) that have traditionally been 
acquired via one-to-one or small-group in-person train- 
ing, methods that effectively limit ability to train mental 
health providers in adequate numbers to address current 
clinical needs. Moreover, this study addresses the import- 
ant comparative effectiveness analysis initiative put forth 
by the U.S. Federal Government [33], which promotes 
the rigorous evaluation of different interventions to en- 
sure that patients receive the most effective and cost effi- 
cient medical care. Recognizing that effective diffusion of 
robust evidence-based interventions into everyday prac- 
tice is as essential as development of successful interven- 
tions themselves [34], the approach was developed to 
allow maximum transportability and accessibility for re- 
search or training purposes. To ensure evaluation of a 
supervision model that is maximally transportable, we 
chose not to include audio or video recording of partici- 
pant clinical sessions for review by supervisors. Although 
such review is likely to improve the quality of supervi- 
sion, it requires time and effort on the part of providers 
and supervisors that raises the burden of the training 
process and may be difficult to implement widely on a 
routine basis. The trial itself combines elements of effect- 
iveness {e.g., participants were providers operating in 



routine client care settings, supervision procedures were 
designed to be easy to implement and limited in terms of 
their time and effort burden) and efficacy research (e.g., 
careful measurement of skills improvement, random as- 
signment to training conditions). We plan to report ana- 
lyses of both intent-to-train and completer analyses, with 
the latter providing more information about the efficacy of 
the training interventions. 

Training addressed three core techniques for achieving 
behavior change: motivational interviewing, goal-setting, 
and behavioral task assignment. The three techniques 
were selected as applying broadly across multiple prob- 
lem areas. We developed an online self-report assess- 
ment of knowledge and perceived self-efficacy and a 
telephone-administered standardized patient simulated 
treatment interview methodology for technique assess- 
ment. Study design allowed an independent evaluation of 
the effects of web-based training, as well as web-based 
training plus supervision delivered by an experienced 
cognitive-behavioral supervisor. Our study is also in- 
novative in the development and utilization of standar- 
dized patients to measure acquisition of treatment 
techniques. Standardized patient methodology is widely 
used in assessing clinical diagnostic techniques in other 
areas of medicine [35,36], but this approach has been rarely 
used as an outcome measure in studies of training in men- 
tal health treatment methods. Our multi-dimensional as- 
sessment design facilitated our collection of systematic 
data across domains of assessment. 

The study included relatively few exclusion criteria. 
We intentionally sought to optimize external validity and 
generalizability to the broadest population of mental 
health providers. Participants consisted of mental health 
providers from a variety of disciplines, including psychol- 
ogists, social workers, nurse clinicians, psychiatrists, and 
others, with varying levels of familiarity with CBT 
methods. 

It is important to recognize that while this training was 
specifically developed for VHA providers, this approach 
to providing internet-based training and telephone 
supervision seems feasible to offer to other groups. Use 
of volunteer research participants is a potential limita- 
tion of the study. However, all trials that consent and 
randomize individuals run the risk of volunteer bias. 

Other design aspects were also aimed to increase 
generalizability, as recommended by Tunis et al. [37] 
and Glasgow et al. [38] for conducting 'practical clinical 
trials' (PCT) in areas of high public health need [3]. In 
addition to recruiting participants broadly representative 
of the target training population, the representativeness 
of organizational settings in which the study is conducted 
is a component of PCT design. The range of organizational 
settings in the present study was intentionally broad and 
included mental health clinics and non-medical center 
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programs such as community-based 'vet centers.' A limita- 
tion, however, is that not all VHA clinical settings could be 
included, and it should be noted that many veterans con- 
tinue to receive health care outside of VHA facilities. 

A final PCT principle concerns the use of broad and 
clinically-relevant outcome measures [37,38] . We attempted 
to assess a range of outcomes (knowledge, perceived self- 
efficacy, self-reported implementation, independent rating 
of technique competence) possible for a study of this type. 
This was a major focus of our study design. On the other 
hand, we were required to make choices as to the selection 
of measures that will inevitably limit our conclusions. Our 
focus on clinical technique assessment via standardized 
patient interviews, in particular, was a critical choice for 
outcomes assessment, the results of which will be evident 
when the study is unblinded. There are some significant 
limitations to our assessment procedures. Most of the as- 
sessment measures used were study-specific, and have not 
been validated. We were unable to locate, for example, 
established measures of provider knowledge that mapped 
onto the techniques we targeted. We constructed simple 
self-efficacy ratings linked to elements of the techniques. 
Because we focused on limited aspects of motivational 
interviewing, we did not use existing motivational inter- 
viewing measurement instrumentation. We also did not in- 
clude measures of patient outcomes related to the training 
content (e.g., rates of engagement in the PTSD treatment 
process as an indicator of the effectiveness of training in 
motivational interviewing). This was judged too difficult to 
achieve given the resources of the study, but should be 
included whenever possible in studies of this type. Overall, 
in this study, we attempted to balance traditional know- 
ledge-based measures with more potentially valid 'cutting 
edge' measures involving standardized patient interviews 
and behavioral ratings of clinical techniques [39-41]. 

Conclusions 

The study described is the first to prospectively evaluate 
in a controlled, randomized design, the effects of web- 
based training in core techniques in cognitive-behavioral 
therapy for veterans with PTSD. The urgent need for 
practitioners qualified to deliver such treatment high- 
lights the need to develop evidence-based training meth- 
ods for changing practice, knowledge, and self-efficacy 
among mental health providers both in large healthcare 
systems and smaller practices and clinics. Our study will 
examine the separate but potentially interactive contri- 
butions of web-based and non-web based training com- 
ponents, in addition to specific approaches to assessment 
of knowledge gain, technique development, and per- 
ceived self-efficacy changes that occur following each of 
the training interventions. Few previous studies have 
employed such a rigorous and well-controlled design for 
assessing training effects in mental health providers 



serving those returning from military deployments: an 
area of extremely high public health need [3]. 
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